Pixel analysis and frame alignment for background frames

ABSTRACT

Background frames can be completed from a collection of frames having foreground objects that are partially obscuring the pixels comprising the background. The special offset of a pixel represented across a collection of frames can be determined based on camera movement data. By determining the relative offset of a pixel represented in a first frame from the same pixel in other frames, pixel values representing a background object can be accumulated to derive a completed background frame.

TECHNICAL FIELD

Techniques of video editing are disclosed, specifically, techniques forcompleting a background frame from a collection of pictures or framesare disclosed.

INTRODUCTION

Over the last decade significant advances have been made in video objectmanipulation and analysis. Special effects in movie and picture editingand even film generation have been just some of the fields that havebenefited from these advancements. Even techniques developed forartificial intelligence and machine vision have been borrowed to enhancevideo object manipulation and analysis. However, even with theseadvancements video editing is still limited by a two dimensionalworkspace that often limits the data that can be captured.

One such example is in working with objects that wholly or partiallyobscure the background of a picture or video frame. Due to thelimitations of a two-dimensional workspace, no pixel data representingthat background can be recorded. While this problem is typically solvedduring the creation of the original video by filming an entirebackground frame without any foreground action, this technique is notavailable to after-the-fact editing or manipulation. Accordingly, amethod for completing a background frame from picture or video frames isneeded.

Currently technology attempts to blend blank pixels with the rest of thebackground, or repeat pixels from elsewhere in the same frame, but thesetechniques have obvious flaws in that they are often noticeable and notof sufficient quality. Accordingly, a more advanced method forcompleting a background frame that overcomes the limitations in the artis needed.

SUMMARY

Additional features and advantages of the concepts disclosed herein areset forth in the description which follows, and in part will be obviousfrom the description, or may be learned by practice of the describedtechnologies. The features and advantages of the concepts may berealized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims. These and otherfeatures of the described technologies will become more fully apparentfrom the following description and appended claims, or may be learned bythe practice of the disclosed concepts as set forth herein.

The present disclosure describes methods and arrangements for completinga background plate or a background frame from a collection of frameswherein a foreground object at least partially obscures the background.The method comprises selecting a first frame and one or more secondframes from a collection of frames for analysis. By analyzing thecollection of frames, data from pixels that represent a portion of thebackground can be collected and combined into a complete backgroundframe.

In order to track pixels across frames, (and identify the correct pixelsvalues to complete the background) the spatial offset between the pixelsrepresenting the first and second frames to be analyzed is determined.The spatial offset is determined by analyzing frames to identify atleast two clusters of pixels having little inter-frame change anddetermining the relative movement of at least one cluster of pixelsagainst at least one other cluster of pixels. From this data, the pixelsof the second frame can be located from the positions of the pixelsmaking up the first frame. In some embodiments the frames themselves canbe aligned so that the pixels representing a first frame will directlymap to the pixels representing a second frame.

Pixels making up the frames are analyzed to determine their pixelvalues. The respective pixel values for each of the pixels are analyzedacross the frames and the predominant pixels values are retained tocomplete the background frame.

In some embodiments, foreground and background objects can be identifiedand this information can be used to determine which pixel values toretain for completion of the background. Foreground and backgroundobjects can be identified by calculating vectors for a plurality ofpixels represented in the first frame and the one or more second frames.The vector represents the respective distance a pixel has moved from oneframe to the next due to a change in camera position. Based on themagnitude of the vector, foreground and background objects can bedistinguished. Objects comprised of pixels having greater vectors can beidentified as foreground objects and objects comprised of pixels havinglesser vectors can be identified as background objects.

In some embodiments a pixel map of an object that appears in at leasttwo frames of a video but is partially obscured in each frame by aforeground object can be completed by calculating an offset betweenpixels of different frames and recording pixel values for pixelsunobscured by the foreground object from each frame, and completing theobject by retaining pixel values that are not representative of theforeground object.

In some embodiments it is further useful to detect edges of theforeground object, and discard all pixel values contained within anoutline of the foreground object comprised by the edges of theforeground object.

Also disclosed are devices for carrying out the above method. Similarly,the described embodiments can all be recorded on a computer-readablemedium having computer readable instructions stored thereon and usefulfor instructing various processor-based devices for carrying out themethods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to best describe the manner in which the above-describedembodiments are implemented, as well as define other advantages andfeatures of the disclosure, a more particular description is providedbelow and is illustrated in the appended drawings. Understanding thatthese drawings depict only exemplary embodiments of the invention andare not therefore to be considered to be limiting in scope, the exampleswill be described and explained with additional specificity and detailthrough the use of the accompanying drawings in which:

FIG. 1 illustrates an example computing device;

FIG. 2 illustrates an example frame having a partially obscuredbackground and a camera movement analysis embodiment;

FIG. 3 illustrates a method embodiment for completing a backgroundframe;

FIG. 4 illustrates a frame repositioned relative to a first frame;

FIG. 5 illustrates a background frame object temporarily obscured;

FIG. 6 illustrates a background frame completion embodiment;

FIG. 7 illustrates a completed background frame; and

FIG. 8 illustrates an extracted foreground object.

DETAILED DESCRIPTION

Various embodiments of the disclosed methods and arrangements arediscussed in detail below. While specific implementations are discussed,it should be understood that this is done for illustration purposesonly. A person skilled in the relevant art will recognize that othercomponents, configurations, and steps may be used without parting fromthe spirit and scope of the disclosure.

With reference to FIG. 1, a general-purpose computing device 100 whichcan be portable or stationary is shown, including a processing unit(CPU) 120 and a system bus 110 that couples various system componentsincluding the system memory such as read only memory (ROM) 140 andrandom access memory (RAM) 150 to the processing unit 120. Other systemmemory 130 may be available for use as well. It can be appreciated thatthe system may operate on a computing device with more than one CPU 120or on a group or cluster of computing devices networked together toprovide greater processing capability. The system bus 110 may be any ofseveral types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. A basic input/output (BIOS) stored in ROM 140 or thelike, may provide the basic routine that helps to transfer informationbetween elements within the computing device 100, such as duringstart-up. The computing device 100 further includes storage devices suchas a hard disk drive 160, a magnetic disk drive, an optical disk drive,tape drive or the like. The storage device 160 is connected to thesystem bus 110 by a drive interface. The drives and the associatedcomputer readable media provide nonvolatile storage of computer readableinstructions, data structures, program modules and other data for thecomputing device 100. In one aspect, a hardware module that performs aparticular function includes the software component stored in a tangiblecomputer-readable medium in connection with the necessary hardwarecomponents, such as the CPU, bus, display, and so forth, to carry outthe function. The basic components are known to those of skill in theart and appropriate variations are contemplated depending on the type ofdevice, such as whether the device is a small, handheld computingdevice, a desktop computer, or a large computer server.

Although the exemplary environment described herein employs a hard disk,it should be appreciated by those skilled in the art that other types ofcomputer readable media which can store data that is accessible by acomputer, such as magnetic cassettes, flash memory cards, digitalversatile disks, cartridges, random access memories (RAMs), read onlymemory (ROM), a cable or wireless signal containing a bit stream and thelike, may also be used in the exemplary operating environment.

To enable user interaction with the computing device 100, an inputdevice 190 represents any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. The input maybe used by the presenter to indicate the beginning of a speech searchquery. The device output 170 can also be one or more of a number ofoutput mechanisms known to those of skill in the art. For example, videooutput or audio output devices which can be connected to or can includedisplays or speakers are common. Additionally, the video output andaudio output devices can also include specialized processors forenhanced performance of these specialized functions. In some instances,multimodal systems enable a user to provide multiple types of input tocommunicate with the computing device 100. The communications interface180 generally governs and manages the user input and system output.There is no restriction on the disclosed methods and devices operatingon any particular hardware arrangement and therefore the basic featuresmay easily be substituted for improved hardware or firmware arrangementsas they are developed.

For clarity of explanation, the illustrative system embodiment ispresented as comprising individual functional blocks (includingfunctional blocks labeled as a “processor”). The functions these blocksrepresent may be provided through the use of either shared or dedicatedhardware, including, but not limited to, hardware capable of executingsoftware. For example the functions of one or more processors presentedin FIG. 1 may be provided by a single shared processor or multipleprocessors. (Use of the term “processor” should not be construed torefer exclusively to hardware capable of executing software.)Illustrative embodiments may comprise microprocessor and/or digitalsignal processor (DSP) hardware, read-only memory (ROM) for storingsoftware performing the operations discussed below, and random accessmemory (RAM) for storing results. Very large scale integration (VLSI)hardware embodiments, as well as custom VLSI circuitry in combinationwith a general purpose DSP circuit, may also be provided.

The logical operations of the various embodiments are implemented as:(1) a sequence of computer implemented steps, operations, or proceduresrunning on a programmable circuit within a general use computer, (2) asequence of computer implemented steps, operations, or proceduresrunning on a specific-use programmable circuit; and/or (3)interconnected machine modules or program engines within theprogrammable circuits.

The present system and method is particularly useful for separating outobjects within a video or picture frame using information from at leastone other video or picture frame. See for example, FIG. 2, whichillustrates a photograph of a girl 202 in front of a mountain 204. Usingthe present system and method, it is possible to complete the obscuredportions of the mountain 204 to create a more complete background frame.Furthermore, the present system and method can separate the girl 202,from the background 204 and move the foreground image independently ofthe background.

The process fills in the obscured pixels of background objects, as shownin FIG. 3. The process identifies other frames from the same scene as atarget frame that can be analyzed to borrow pixels needed to fill inobscured areas 302. From a group of candidate frames (FIGS. 2, 4, 5, forexample), the process can compare a target frame to at least one otherframe. While one other frame might be sufficient to complete thebackground frame, according to the method described herein, largergroups of candidate frames increase the likelihood that the neededpixels can be discovered.

For all frames to be compared, the process must determine the relativemotion of the camera used to take the photograph or record the video. Ifthe image was recorded using a digital camera, metadata may be availablepertaining to the frame positioning corrective data associated with analgorithm for that purpose, for example, anti-shake technology 305. Ifthere is no camera metadata, the relative movement of camera betweenframes can be determined by comparing the motion of one group of pixelsto another group of pixels across the frames to be analyzed. Forexample, see FIG. 2 wherein groupings of pixels 210 can be compared withgroupings of pixels 212 across multiple frames. Based on the movement ofpixels 212 relative to pixels 210, camera motion (in the x,y plane)between the two pixels can be determined. Motion in the z-plane can bedetermined by comparing the pixel density of background imagery to thepixel density of foreground imagery. In some embodiments camera motioncan be analyzed using metadata from the camera itself.

Where the relative motion of pixel clusters is required to determinecamera movement, the process can select clusters of pixels to use in theanalysis. The process can analyze across the group of candidate framesfor pixel areas that have little inter-frame change. In someembodiments, clustering algorithms can be used to identify commonclusters of pixels across the group of candidate frames 307. Theclusters can be of any number of pixels sufficient to easily trackacross the candidate frames, however in at least some embodiments, it ispreferred that the clusters of pixels be at least 10×10 pixels, 13×13pixels, or 20×20 pixels. In some embodiments, the process selects pixelsnear the outer portions of the respective frames having good contrast tothe surrounding areas. In some embodiments, pixels making up an objectof focus can also be used to determine the relative motion of the camerafrom one frame to the next. In some embodiments, a greater number ofpoints can be selected, or multiple techniques can be combined.

If however, any of the pixels clusters become obscured or out of framein any of the candidate frames, the process can repeat the motionanalysis by selecting new groups of pixels. The new groups of pixels donot necessarily need to be represented in every frame, but at a minimum,the groups of pixels must be able to determine the motion of the camerarelative to the camera's motion in another already analyzed frame.

In some embodiments, once the camera motion has been detected, theframes that are to be analyzed need to be repositioned in the x, yplanes to eliminate the camera motion 314. This process serves toeffectively make it so that if any two frames were to be overlaid uponeach other, the pixels in the frames would align. Because of thisrelationship, it is possible to accurately identify and track pixelsacross multiple frames. In some embodiments, the frame repositioningdoes not require a physical repositioning. Using the data correspondingto the camera motion across the frames, the method calculates a spatialoffset corresponding to the offsetting location of a pixel in one framecompared to the location of the same pixel in another frame. Using thistechnique the method can identify pixels across frames that representthe same portion of the background object.

Each pixel is identified across the frames to be analyzed and the pixelvalues for each pixel are sampled 318. In the simplest embodiments,pixels can be tracked because they directly correspond to each otherfrom frame to frame after repositioning the frames 314. However, severaltechniques are known in the art for tracking a pixel's movement within aframe. For example, the process can look for similar pixels within arange of 20×20 pixels from the original position of a target pixel.Additionally, the process can take in account possible camera motion inthe z-plane and detect pixels that are now merged or divided intoseveral separate pixels due to zoom, or z-plane camera movement 309.

In sampling for any given pixel in any given frame, a value is obtained.For example, for a pixel 30,30 it may have values 0, 200, 20 (RBG) inthe first sample frame, and values 0, 202, 19 in a second frame andvalues 200, 100, 25 in a final frame. The values in the first two framesare nearly identical and indicate that the particular analyzed pixel isrepresenting the same object in the first two frames. While the pixelvalues differ slightly, they still likely represent the same portion ofthe image in each of the frames and the difference in values can beattributed to lighting, exposure or other minor variations. However, thevalues for the same pixel in the third frame differ enough that thepixel is likely representing a different object.

In some embodiments, the method can accommodate variations in pixelvalues of up to 15% and still identify them as representing the sameobject. Pixel values having a greater variation run the risk ofrepresenting a different object in the frame and these variations areoutside the range of likely inter-frame variation for the same object.

Analyzing the range of values for each pixel, the process maintains themost predominant values 320 and assumes that this value is the value ofthe background object. This assumption is made because a backgroundobject is likely to remain more consistent and is less likely to moveabout the frame, as would the object of focus.

This is best illustrated with reference to FIGS. 4 and 5. In FIG. 5, thegirl's hand obscures a bush in the background (see view box 220). Forthis frame, a value would be a sample that is different than the valuesampled in FIG. 4 or FIG. 2, which are not obscured by the girl's hand.Thus, in this example, by maintaining the most predominant value, theprocess will correctly choose pixel values for the background image and,over the course of several frames, will complete a background frame.

FIG. 6 illustrates the effect of maintaining the predominant pixels. Inthis figure, half of the girl has been replaced with pixel values of thebackground. This figure illustrates the application of the processexplained above. Across the frames selected for analysis by this method,the process was able to identify predominant (and consistent) values forthe filled in area because in the other frames analyzed to develop thisfigure the girl had moved and exposed the background. FIG. 7 illustratesthe same frame after the process has analyzed additional frames and hasextracted the complete background frame. In this figure only thebackground remains; any pixels that had represented the girl in thisframe have been replaced with pixels representing the background.

In some embodiments, additional analysis can be performed to eliminatethe reliance of the aforementioned presumption that the most predominantvalues represent the background. For example, the process can computepixel motion vectors to determine the relative amounts of inter-framemovement 322. Using a method similar to that described by Lucas, B. andKanade, T., “An Iterative Image Registration Technique with anApplication to Stereo Vision” Proc. 7^(th) Intl Conf on ArtificialIntelligence (IJCAI) 1981, Aug. 24-28, pp. 674-679, pixel vectors can becomputed and based on the relative vector of a given pixel compared toothers in the frame and corresponding depth values can be determined. Apixel representing a portion of a foreground object will have a greatervector than a pixel representing a portion of a background object. Usingthese vector values, pixels representing a background object can beascertained and retained 324.

Taking the analysis one step further, pixel values that might onlyappear in one frame (thus having no vector) but representing a portionof the background can be determined by an analysis of these pixelsvalues against the pixel values known to represent a foreground objectand background object. The relative similarity of the unknown pixels tothe known pixels can be used to determine whether these pixel valuesshould be retained as a missing piece of a background.

This type of analysis can be especially useful when considering only twoframes. Across two frames, any pixels that are obscured in one frame,but not in another will not have any vector values. However, otherportions of the background objects and foreground objects will have beenknown to be or assumed to represent the background. Thus, the transientportions of the background can be compared with the values of the pixelsalready determined to represent the background and the pixels having thevalues most similar to the rest of the background are retained.

In some embodiments, it may also be useful to compare these unassociatedpixels with values know to represent the foreground image as well.However, this comparison would select the pixels least likely torepresent a portion of the foreground. This second layer of analysis canadd an extra degree of confidence that the proper pixels are retained.

In other embodiments, the vector analysis is further useful to extract aforeground object or reposition the foreground object. By performing avector analysis, the pixels making up the foreground object will havethe greatest vector and thus the majority of the foreground object canbe identified in this manner. The rest of the object can be extractedusing known edge detection algorithms to trace the edge of theforeground object. See for example FIG. 8 wherein the girl has beenextracted from the background.

Having completed a background frame and extracted a foreground object,the foreground object can be manipulated and moved within the frameseparately from the background.

It will be appreciated that across the various embodiments it ispossible that the entire background frame will not be completed if theforeground object continues to obscure the background. In such cases, nopixel values for the background can be identified and incorporated intothe background frame. Such eventuality becomes less likely as the numberof frames analyzed increases. However, in situations where pixel datarepresenting the background image cannot be obtained, these pixels canbe manufactured using less satisfactory methods known in the art.

Embodiments within the scope of the present invention may also includecomputer-readable media for carrying or having computer-executableinstructions or data structures stored thereon. Such computer-readablemedia can be any available media that can be accessed by a generalpurpose or special purpose computer. By way of example, and notlimitation, such tangible computer-readable media can comprise RAM, ROM,EEPROM, CD-ROM or other optical disk storage, magnetic disk storage orother magnetic storage devices, or any other medium which can be used tocarry or store desired program code means in the form ofcomputer-executable instructions or data structures. Combinations of theabove should also be included within the scope of the tangible orintangible computer-readable media.

Computer-executable instructions include, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. Computer-executable instructions also includeprogram modules that are executed by computers in stand-alone or networkenvironments. Generally, program modules include routines, programs,objects, components, and data structures that perform particular tasksor implement particular abstract data types. Computer-executableinstructions, associated data structures, and program modules representexamples of the program code means for executing steps of the methodsdisclosed herein. The particular sequence of such executableinstructions or associated data structures represent examples ofcorresponding acts for implementing the functions described in suchsteps.

Those of skill in the art will appreciate that other embodiments of theinvention may be practiced in network computing environments with manytypes of computer system configurations, including personal computers,hand-held devices, multi-processor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. Embodiments may also be practiced indistributed computing environments where tasks are performed by localand remote processing devices that are linked (either by hardwiredlinks, wireless links, or by a combination thereof) through acommunications network. In a distributed computing environment, programmodules may be located in both local and remote memory storage devices.

Communication at various stages of the described system can be performedthrough a local area network, a token ring network, the Internet, acorporate intranet, 802.11 series wireless signals, fiber-optic network,radio or microwave transmission, etc. Although the underlyingcommunication technology may change, the fundamental principlesdescribed herein are still applicable.

The various embodiments described above are provided by way ofillustration only and should not be construed to limit the invention.For example, the principles herein may be applied to an online storeaccessible wirelessly by a portable media playback device or by apersonal computer physically connected to a network. Those skilled inthe art will readily recognize various modifications and changes thatmay be made to the present invention without following the exampleembodiments and applications illustrated and described herein, andwithout departing from the true spirit and scope of the presentdisclosure.

1. A method for completing a background frame from two or more frames ina collection comprising: selecting a first frame and one or more secondframes from a collection of frames for analysis; determining a spatialoffset between the first and second frames; aligning pixels of thesecond frames with the pixels of the first frame based on the spatialoffset data; analyzing the pixels comprising the first and secondframes; and maintaining predominant pixels values across the analyzedframes.
 2. The method of claim 1, wherein the special offset isdetermined by analyzing frames to identify at least two clusters ofpixels having little inter-frame change and determining the relativemovement of at least one cluster of pixels against at least one othercluster of pixels.
 3. The method of claim 1, further comprising:calculating vectors for a plurality of pixels represented in the firstframe and the one or more second frames, the vector representing therespective distance a pixel has moved from one frame to the next due toa change in camera position; and identifying objects comprised of pixelshaving greater vectors as foreground objects and objects comprised ofpixels having lesser vectors as background objects.
 4. The method ofclaim 3, further comprising: averaging the pixel values of those pixelsidentified as background objects and comparing pixel values from otherframes against the average; identifying pixels from other frames havingapproximately similar pixel values as also comprising background pixels;and maintaining these pixel values across the analyzed frames.
 5. Themethod of claim 1, wherein the frames comprise individual photographs.6. A non-transitory computer-readable medium storing computer executableinstructions for implementing a method on a device, the instructionscomprising: selecting a first frame and one or more second frames from acollection of frames for analysis; determining a spatial offset betweenthe first and second frames; aligning pixels of the second frames withthe pixels of the first frame based on the spatial offset data;analyzing the pixels comprising the first and second frames; andmaintaining predominant pixels values across the analyzed frames.
 7. Thenon-transitory computer-readable medium of claim 6, wherein the specialoffset is determined by analyzing frames to identify at least twoclusters of pixels having little inter-frame change and determining therelative movement of at least one cluster of pixels against at least oneother cluster of pixels.
 8. The non-transitory computer-readable mediumof claim 6, the instructions further comprising: calculating vectors fora plurality of pixels represented in the first frame and the one or moresecond frames, the vector representing the respective distance a pixelhas moved from one frame to the next due to a change in camera position;and identifying objects comprised of pixels having greater vectors asforeground objects and objects comprised of pixels having lesser vectorsas background objects.
 9. The non-transitory computer-readable medium ofclaim 8, the instructions further comprising: averaging the pixel valuesof those pixels identified as background objects and comparing pixelvalues from other frames against the average; identifying pixels fromother frames having approximately similar pixel values as alsocomprising background pixels; and maintaining these pixel values acrossthe analyzed frames.
 10. A device comprising: a video processorconfigured to select a first frame and one or more second frames from acollection of frames for analysis and determining a spatial offsetbetween the first and second frames, the processor further configuredfor analyzing pixels comprising the first and second frames andcomparing values corresponding to a pixel in a first frame with a pixelin the second frame, the pixel in the second frame being identifiedbased on the special offset with respect to the location of the firstpixel in the first frame, and configured to maintain predominant pixelsvalues across the analyzed frames.